Skip to content

Azure: Create clusters with no public endpoints #2521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-merge-robot merged 1 commit into from
Oct 30, 2019
Merged

Azure: Create clusters with no public endpoints #2521

openshift-merge-robot merged 1 commit into from
Oct 30, 2019

Conversation

@jhixson74
Copy link
Member

@jhixson74 jhixson74 commented Oct 17, 2019

This gives the install the ability to install an OpenShift cluster on Azure as
internal/private, which is only accessable from an internal network and not
visible on the Internet.

https://jira.coreos.com/browse/CORS-1227

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 17, 2019
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from 91406ea to 0c2fe22 Compare October 17, 2019 18:52
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 17, 2019

/test e2e-azure

patrickdillon
patrickdillon reviewed Oct 17, 2019
View reviewed changes
@ironcladlou ironcladlou mentioned this pull request Oct 17, 2019
Merged
2 tasks
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from 0c2fe22 to 4ce5271 Compare October 17, 2019 20:30
@ironcladlou
Copy link
Contributor

ironcladlou commented Oct 18, 2019

@abhinavdahiya @crawford @jhixson74

Would it be useful to tease apart the Azure-ness of this PR and extract the new generic PublishingStrategy support into a separate PR? Once #2523 lands I would like to use the new field for an ingress PR.

Or, any feeling for when this PR will merge?

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 18, 2019

@ironcladlou 31a9d52 is a little fleshed out version of publish, and #2526 is only waiting for review from @wking

So as soon as #2526 lands you can start looking at it. Also fell free to take that commit and work up a PR to unblock too.

@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch 3 times, most recently from 6c9def9 to 8291f56 Compare October 22, 2019 00:50
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 22, 2019
View reviewed changes
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch 3 times, most recently from de566da to b2c84c9 Compare October 22, 2019 05:43
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 22, 2019

/test e2e-azure

@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from b2c84c9 to ae4754f Compare October 24, 2019 20:34
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 24, 2019

/test e2e-azure

abhinavdahiya
abhinavdahiya reviewed Oct 24, 2019
View reviewed changes
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from ae4754f to 5d569de Compare October 24, 2019 21:12
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 24, 2019

/test e2e-azure

@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from 5d569de to 8b0bdaa Compare October 25, 2019 01:48
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 25, 2019

/test e2e-azure

@jhixson74 jhixson74 changed the title Azure: [WIP] Create clusters with no public endpoints Azure: Create clusters with no public endpoints Oct 25, 2019
abhinavdahiya
abhinavdahiya reviewed Oct 25, 2019
View reviewed changes
abhinavdahiya
abhinavdahiya reviewed Oct 25, 2019
View reviewed changes
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch 2 times, most recently from 3254aa4 to b431d6d Compare October 28, 2019 18:27
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 28, 2019

/test e2e-azure

@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from b431d6d to caaaf08 Compare October 28, 2019 20:28
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 28, 2019

/test e2e-azure

1 similar comment
@jhixson74
Copy link
Member Author

jhixson74 commented Oct 28, 2019

/test e2e-azure

@jhixson74
Copy link
Member Author

jhixson74 commented Oct 28, 2019 •
edited
Loading

Tested and confirmed working with the following resources:

resource group: jhixson_vnet_rg
virtual network: jhixson_test_vnet (10.0.0.0/16)
subnet: jhixson_test_master_subnet (10.0.0.0/19)
subnet: jhixson_test_worker_subnet (10.0.32.0/19)
network security group: jhixson_test_master_nsg (22, 6443)
network security group: jhixson_test_worker_nsg (22, 80, 443, 6443)

I created a bastion host on the control plane network (jhixson_test_master_subnet) and setup a VPN server. It is necessary to connect the install client to the VPN server and have DNS configured for this to work. sshuttle can possibly be used for an easier configuration. You will need access to the vnet networks for this to be successful.

VPN server config:

port 1194
proto udp
dev tun
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/openvpn-server.crt
key /usr/local/etc/openvpn/keys/openvpn-server.key
dh /usr/local/etc/openvpn/keys/dh.pem
remote-cert-tls client
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 10.0.0.0 255.255.224.0"
push "route 10.0.32.0 255.255.224.0"
push "route 168.63.129.16 255.255.255.255"
keepalive 10 120
tls-auth /usr/local/etc/openvpn/keys/ta.key 0
cipher AES-256-CBC
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3
explicit-exit-notify 1

VPN client config:

client
dev tun
proto udp
remote x.x.x.x 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /home/jhixson/openvpn/keys/ca.crt
cert /home/jhixson/openvpn/keys/jhixson.remote.csb.crt
key /home/jhixson/openvpn/keys/jhixson.remote.csb.key
remote-cert-tls server
tls-auth /home/jhixson/openvpn/keys/ta.key 1
cipher AES-256-CBC
verb 3

/etc/resolv.conf:

search redhat.com remote.csb 0bdxka02uqnuvpajnzo52ntqfd.gx.internal.cloudapp.net
nameserver 168.63.129.16
nameserver 127.0.0.1

Here is the relevant install-config.yaml for a private/internal cluster:

apiVersion: v1
baseDomain: installer.azure.devcluster.openshift.com
publish: Internal
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    azure:
      osDisk:
        diskSizeGB: 128
      type: Standard_D4s_v3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: testprivate
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  azure:
    baseDomainResourceGroupName: os4-common
    region: centralus
    networkResourceGroupName: jhixson_vnet_rg
    virtualNetwork: jhixson_test_vnet
    controlPlaneSubnet: jhixson_test_master_subnet
    computeSubnet: jhixson_test_worker_subnet

Tested and confirmed working for external/public clusters also.

install-config.yaml for external/public cluster:

apiVersion: v1
baseDomain: installer.azure.devcluster.openshift.com
compute:
- hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: jhixson42
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  azure:
    baseDomainResourceGroupName: os4-common
    region: centralus

@patrickdillon patrickdillon mentioned this pull request Oct 29, 2019
Merged
@jhixson74
Azure: Create clusters with no public endpoints
442395f 
This gives the install the ability to install an OpenShift cluster on Azure as
internal/private, which is only accessable from an internal network and not
visible on the Internet.

Because of limitations in Azure, VM's without a public IP address behind a load
balancer without a public IP address are unable to reach the internet. To get
around this limitation, a dummy load balancer with a public IP address is setup
with a rule to a dummy service.

https://jira.coreos.com/browse/CORS-1227
@jhixson74 jhixson74 force-pushed the master_azure_no_public_endpoints branch from caaaf08 to 442395f Compare October 29, 2019 01:53
@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 29, 2019

/test e2e-azure

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 29, 2019

/approve

waiting for e2e-azure to go green before dropping lgtm

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 29, 2019 •
edited
Loading

@jhixson74: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-openstack 442395f link /test e2e-openstack
ci/prow/e2e-aws-scaleup-rhel7 442395f link /test e2e-aws-scaleup-rhel7

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jhixson74
Copy link
Member Author

jhixson74 commented Oct 29, 2019

/test e2e-azure

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 29, 2019

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 29, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, jhixson74

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit d64b4d0 into openshift:master Oct 30, 2019
@jhixson74 jhixson74 mentioned this pull request Nov 26, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@patrickdillon patrickdillon patrickdillon left review comments

@mtnbikenc mtnbikenc Awaiting requested review from mtnbikenc

+1 more reviewer

@abhinavdahiya abhinavdahiya abhinavdahiya left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@abhinavdahiya abhinavdahiya

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@jhixson74 @ironcladlou @abhinavdahiya @openshift-ci-robot @patrickdillon @openshift-merge-robot